Skip to content

fix(live): keep trace git metadata and late spans visible#1134

Merged
XinweiHe merged 9 commits into
traceroot-ai:mainfrom
RitwijParmar:codex/traceroot-live-trace-reliability
Jul 11, 2026
Merged

fix(live): keep trace git metadata and late spans visible#1134
XinweiHe merged 9 commits into
traceroot-ai:mainfrom
RitwijParmar:codex/traceroot-live-trace-reliability

Conversation

@RitwijParmar

@RitwijParmar RitwijParmar commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

References #1099 and #1006.

This fixes two live-trace edge cases that come from treating the root span as the whole trace lifecycle.

What changed

  • Promote traceroot.git.repo and traceroot.git.ref from child spans into the trace record as soon as they arrive, instead of waiting for the root span. The root span still wins if it carries a value later.
  • Keep the live SSE stream open for a short quiet window after a root-complete signal. This lets late descendant span batches arrive before the frontend receives trace_complete and closes the stream.
  • Preserve the subscribe-before-ClickHouse-check invariant so concurrent Redis span events are still not missed.
  • Add regression coverage for child-before-root git metadata, root-over-child precedence, root-without-git fallback, already-complete quiet completion, and late spans arriving after trace_complete.

Why this matters

For live agent runs, the trace can be useful before the root span lands or before all descendants have arrived. This makes repo/ref visible earlier for debugging and prevents the UI from closing the stream while the trace is still growing.

Validation

  • uv run ruff check backend/rest/routers/live.py backend/worker/otel_transform.py tests/rest/test_live_router.py tests/worker/test_otel_transform.py
  • uv run python -m py_compile backend/rest/routers/live.py backend/worker/otel_transform.py tests/rest/test_live_router.py tests/worker/test_otel_transform.py
  • uv run pytest tests/rest/test_live_router.py tests/worker/test_otel_transform.py -q passed, 98 tests
  • git diff --check

Summary by cubic

Improve live trace reliability by surfacing git repo/ref early and holding the SSE stream open for a short quiet window after completion. The window is anchored to the latest of the root’s end time and the last ingest activity so late spans are delivered while old completed traces close immediately.

  • Bug Fixes
    • Promote traceroot.git.repo and traceroot.git.ref from child spans into the trace as they arrive; root values overwrite, and child values persist if the root has none.
    • Emit trace_complete after a quiet window triggered by ClickHouse root completion or a Redis trace_complete; reset on late spans or redundant trace_complete; anchor the deadline to max(root end, last ingest) and close immediately if it’s already past; subscribe to Redis before the ClickHouse check and forward spans received during the check/window before closing; at the hard ceiling emit trace_complete if the root completed, otherwise stream_timeout; the window length is settings.trace_complete_quiet_seconds (default 10s, above the SDK flush interval).

Written for commit ce0261f. Summary will update on new commits.

Review in cubic

@RitwijParmar
RitwijParmar requested a review from a team as a code owner June 10, 2026 07:54

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Client as Frontend Client
    participant SSE as Live SSE Stream (live.py)
    participant Redis as Redis PubSub
    participant CH as ClickHouse
    participant Worker as Background Worker (otel_transform.py)

    Note over Client,Worker: Live Trace SSE Flow - Happy Path

    Client->>SSE: GET /live/{trace_id}
    SSE->>Redis: subscribe(trace_channel)
    Note over SSE,Redis: NEW: Subscribe before ClickHouse check
    
    SSE->>CH: _is_trace_complete_in_clickhouse()
    alt Already complete in ClickHouse
        CH-->>SSE: root span with end_time found
        SSE->>SSE: NEW: start quiet window (TRACE_COMPLETE_QUIET_SECONDS)
        loop Quiet window active
            SSE->>Redis: poll for messages
            alt Late span batch arrives
                Redis-->>SSE: {"type": "spans", ...}
                SSE->>SSE: NEW: reset quiet window deadline
                SSE-->>Client: event: spans / data...
            else Quiet window expires
                SSE-->>Client: event: trace_complete
            end
        end
    else Not yet complete
        CH-->>SSE: no root span end_time
        SSE->>SSE: stream normally, no quiet deadline
    end

    Note over Worker,CH: Span Processing (concurrent)

    Worker->>Worker: transform_otel_to_clickhouse()
    Note over Worker: NEW: Extract git ref/repo from child spans
    Worker->>CH: write spans (including child git metadata)
    Worker->>Redis: publish(trace_channel, {"type": "spans", ...})
    
    alt Root span arrives later
        Worker->>Worker: NEW: Root git values override child values
        Worker->>Redis: publish(trace_channel, {"type": "trace_complete"})
    end

    Note over SSE,Redis: Late span delivery after root complete

    SSE->>Redis: poll (quiet window still active)
    Redis-->>SSE: {"type": "spans", ...}
    SSE->>SSE: NEW: reset quiet window
    SSE-->>Client: event: spans / data...

    Note over SSE,Client: Completion Event
    SSE->>SSE: quiet window fully expired
    SSE-->>Client: event: trace_complete
    SSE->>Redis: unsubscribe & close
    Client->>Client: close SSE stream

    Note over SSE,Redis: Edge Case: Redundant trace_complete
    alt Redis sends redundant trace_complete
        Redis-->>SSE: {"type": "trace_complete"}
        SSE->>SSE: NEW: reset quiet window only (no duplicate emit)
    end
Loading

Re-trigger cubic

Comment thread backend/rest/routers/live.py Outdated
Comment thread backend/rest/routers/live.py Outdated
Comment thread backend/rest/routers/live.py Outdated
@dark-sorceror

Copy link
Copy Markdown
Collaborator

I also noticed you had a similar PR open for git half (#1120). Looks identical, so let's keep this one and close the other when this is merged.

@dark-sorceror
dark-sorceror force-pushed the codex/traceroot-live-trace-reliability branch from 1097ae6 to 069a199 Compare July 9, 2026 17:49
@dark-sorceror

Copy link
Copy Markdown
Collaborator

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review

@dark-sorceror I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Client as Frontend Client
    participant SSE as Live Stream Endpoint
    participant Redis as Redis PubSub
    participant CH as ClickHouse
    participant Worker as OTel Transform Worker

    Note over Worker,CH: Background span ingestion (otel_transform.py)

    Worker->>CH: Write span batch
    alt First span for trace is a child with git metadata
        Worker->>Worker: NEW: Promote traceroot.git.repo/ref to trace record immediately
        Worker-->>CH: Trace record includes git_repo, git_ref (child values)
    else Root span arrives later
        Worker->>CH: Update trace record (root values overwrite if present)
    end

    Note over Client,SSE: Live SSE stream (live.py)

    Client->>SSE: GET /live (subscribe)
    SSE->>Redis: Subscribe to trace channel
    Note over SSE,CH: Ensure subscribe before ClickHouse check (unchanged)

    SSE->>CH: Query root_completion_time for trace
    alt Root end_time exists (already has completed root)
        CH-->>SSE: root_end_time
        SSE->>SSE: NEW: Compute quiet deadline anchored to root end time
        alt Deadline already expired (old trace)
            SSE-->>Client: event: trace_complete (immediate)
        else Deadline in future
            SSE->>Redis: listen for messages (timeout = quiet_remaining or heartbeat)
            loop Until deadline or disconnect
                Redis-->>SSE: Optional late spans
                SSE-->>Client: event: spans (forwarded)
                alt Span received
                    SSE->>SSE: NEW: Reset quiet deadline
                end
                alt trace_complete from Redis (redundant)
                    SSE->>SSE: NEW: Reset quiet deadline (no output)
                end
                opt Heartbeat timeout
                    SSE-->>Client: : heartbeat
                end
            end
            SSE-->>Client: event: trace_complete
        end
    else No root end_time (trace still live)
        CH-->>SSE: null
        SSE->>SSE: No quiet deadline set
        loop Until hard ceiling (MAX_STREAM_SECONDS) or disconnect
            Redis-->>SSE: span event
            SSE-->>Client: event: spans
            alt trace_complete from Redis
                SSE->>SSE: NEW: Set quiet deadline (now + TRACE_COMPLETE_QUIET_SECONDS)
                Note over SSE,Client: Continue listening for late spans
            end
            opt Heartbeat
                SSE-->>Client: : heartbeat
            end
        end
        alt Quiet deadline expired (root completed earlier)
            SSE-->>Client: event: trace_complete
        else Timeout before any completion
            SSE-->>Client: event: stream_timeout
        end
    end

    Note over SSE,Client: Frontend closes stream after trace_complete
Loading

Re-trigger cubic

@dark-sorceror

Copy link
Copy Markdown
Collaborator

LGTM

@XinweiHe
XinweiHe merged commit fa00c9e into traceroot-ai:main Jul 11, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants